Augmented-reality systems often portray digitally enhanced images or other scenes with computer-simulated objects. To portray such scenes, an augmented-reality system sometimes renders both real objects and computer-simulated objects with shading and other lighting conditions. Many augmented-reality systems attempt to seamlessly render virtual objects composited with objects from the real world. To achieve convincing composites, an augmented reality system need to illuminate virtual objects with consistent lighting matching a physical scene.
Despite advances in estimating lighting conditions for digitally enhanced scenes, some technical limitations impede conventional augmented-reality systems from realistically lighting virtual objects and accurately reflecting changes to physical environments. For example, conventional augmented-reality systems cannot quickly render or adjust lighting conditions in real (or near-real) time, alter lighting conditions when a digitally enhanced scene changes, or faithfully capture variation of lighting throughout a scene. Three-dimensional scenes exasperate these technical limitations because each location in three dimensions can receive a different amount of light at a given moment from a full 360-degree range of directions. Both the directional dependence and the variation of light across the scene play an important role when attempting to faithfully and convincingly render synthetic objects into the scene.
For example, some conventional augmented-reality systems cannot realistically portray lighting conditions for a computer-simulated object in real (or near-real) time. In some cases, conventional augmented-reality systems use an ambient-light model (i.e., only a single constant term with no directional information) to estimate the light received by an object from its environment. For example, conventional augmented-reality systems often use simple heuristics to create lighting conditions, such as by relying on mean-brightness values for pixels of (or around) an object to create lighting conditions in an ambient-light model. Additionally, existing augmented-reality systems decode full environmental maps of a low dynamic range (“LDR”) image or a high dynamic range (“HDR”) image as a basis for estimating lighting conditions. Based on the environmental map, such existing systems determine a single lighting parameter for the entire image (e.g., by estimating lighting conditions at a default center of the image). Such an approximation does not capture the directional variation of lighting and can fail to produce a reasonable ambient-lighting approximation under many conditions—resulting in unrealistic and unnatural lighting. Indeed, by applying a single lighting parameter, existing systems can illuminate computer-simulated objects with lighting that conflicts with natural lighting of real objects in the immediate physical vicinity.
In addition to challenges to portraying realistic lighting, in some cases, conventional augmented-reality systems cannot flexibly adjust or change lighting conditions for a particular computer-simulated object in a scene. For instance, some augmented-reality systems determine lighting conditions for a digitally enhanced image as a collective set of objects or (as above) for an image as a whole. Because such lighting conditions generally apply to a set of objects or an entire image, conventional systems either cannot adjust lighting conditions for a particular object or can only do so by redetermining lighting conditions for the entire digitally enhanced image in an inefficient use of computing resources.
Independent of technical limitations affecting the realism or flexibility of lighting in augmented reality, some conventional augmented-reality systems can estimate reflections (but not lighting conditions based on a light source) in real time for a virtual object. For instance, some conventional augmented-reality systems use exposure information to determine a relative brightness of an environment by scanning part of the environment and completing a map of the environment using machine-learning algorithms. By extrapolating a map of the environment, such conventional systems can estimate a reflection for virtual objects within the environment, but cannot determine lighting conditions for such virtual objects based on a real light source within or outside of the environment.